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(54) Communication system and method employing automatic language identification 



(57) A method and systems for identifying the lan- 
guage^) of a text or speech based communication by 
comparing an arbitrary segment within the communica- 
tion with one or more language segments contained 
within at least one language database. Once language 
identification of the communication has been achieved, 
a user is provided with options for the subsequent 



processing and/or retrieval of the communication. User 
options include translation to one or more specified sec- 
ondary languages and/or media conversion (e.g. text- 
to-speech or speech-to-text). The converted communi- 
cation can be accessed through available media retriev- 
al devices (e.g. telephone, facsimile, personal compu- 
ter). 
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Description 

FIELD OF THE INVENTION 

[0001] The present invention is directed generally to s 
the communications field, and more particularly, to com- 
munication systems and methods utilizing language 
identification. 

BACKGROUND OF THE INVENTION io 

[0002] The World Wide Web and the global economy 
have increased message quantity and variety to the 
point where many users are receiving messages in a 
multitude of languages. Users who receive messages is 
in an unfamiliar language may need to have a message 
translated prior to being able to understand it. Further- 
more, mobile individuals must retrieve and manage their 
messages from wherever they happen to be. Therefore, 
the abilily to have a communication presented in a de- 20 
sired language and/or media type, regardless of the lan- 
guage and/or media type of the original or received com- 
munication, is desirable for modem day communication 
systems. A 'communication" for purposes of this disclo- 
sure can be at least a text or spccch-bascd document, 2$ 
record, recording message (e.g. : e-mail, voicemail, vid- 
eomail. etc.), facsimile, video clip, or the like or any com- 
bination thereof, such as those commonly transmitted 
on, by. to, or through electronic communication sys- 
tems, such as : multimedia communication systems like 30 
multimedia messaging systems. "Media" as used herein 
refers to the form of a communication, such as a text (as 
in the case of an email, certain facsimiles, or portions 
thereof, etc.). speech (as in the case of voicemail, the 
speech portion of a videomail or clip, recording, etc.), or 35 
the like. 

[0003] Systems that provide language translation ca- 
pabilities are generally available. For example, US 
5,768,603 issued to Brown, et. al. June 16, 1998, and 
US 5,765,131 issued to Stentiford, et. al., June 9, 1998 40 
are directed to language translation technology. In ad- 
dition, today multimedia message systems routinely 
provide text-to-speech conversion of e-mail messages 
to allow a user to hear their e-mail over the telephone. 
US 5,774,854 issued to Sharman, June 30, 1998, is di- 4S 
rected to a text-to-speech (TTS) system. Speech-to-text 
technology is also available for the conversion of speech 
messages to text. US 5,724,410, issued to Parvulescu 
et a I., March 3, 1998 teaches a method of speech-to- 
text conversion where the source language is already $0 
known. 

[0004] The problem in utilizing traditional language 
translation technology in messaging systems, is that the 
original language (source language) of a message must 
be known prior to the processing (e.g., translation and/ ss 
or media conversion, etc.) of that communication. This 
is especially true when the language of a message could 
be one of many. Some messaging protocols provide for 



message language "tagging" or labeling. However, even 
where a protocol supports a language tag, it may not be 
used appropriately in mixed-language applications. A 
single message may include multiple parts or sections 
in differing languages, whereas a protocol as generally 
provided in the art may only identify a single language. 

SUMMARY OF THE INVENTION 

[0005] The present invention addresses the above 
needs, and in particular, the need for language identifi- 
cation by providing automatic language identification 
within communication systems such as multimedia 
communication systems like multimedia messaging 
systems. Furthermore, and in accordance with the 
present invention, such identification thereby enables 
and/or enhances subsequent processing (e.g., transla- 
tion and/or media-type conversion, etc.) of language 
identified communications. 

[0006] In accordance with the present invention there 
is provided a method for automatically identifying the 
languages of a communication by comparing one or 
more arbitrary segments of the communication with one 
or more segments for a plurality of languages. In accord- 
ance with further aspects of the invention, once the lan- 
guages of the communication is/are recognized, the in- 
vention provides at least three options of processing the 
communication: 

[0007] Translating the language identified of the com- 
munication into at least one different or secondary lan- 
guage using an appropriate translator; 

Converting the language identified communication, 
as it exists in one media-type, directly into another 
media-type based on the identified language; or 
Translating the language identified communication 
into at least one different or secondary language us- 
ing an appropriate translator, and then converting 
the communication into another media-type. 

[0008] In accordance with the present invention there 
is also provided an apparatus for automatically identify- 
ing the language(s) of a communication. Language 
identification is achieved by means of a multi-lingual lan- 
guage recognition module having access to one or more 
language databases for comparing one or more arbi- 
trary segments of the communication with one or more 
segments for a plurality of languages. A translation mod- 
ule can be provided for language translation. The trans- 
lation module can access the database(s) for use in car- 
rying out language translation. The translation module 
can be instructed as to which language to use for trans- 
lating a given communication into. Such instruction can 
be set within a system in the form of user preferences, 
or executed on the fly. The language of a particular com- 
munication is identified automatically, and this informa- 
tion can then be passed to the translation module. Me- 
dia-to-media (e.g., text-to-speech or speech-to-text) 
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conversion modules can also be provided for media 
conversion Such modules could have access to the da- 
tabase^) in order to facilitate conversions once appro- 
priate language is identified. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0009] These and other aspects of the present inven- 
tion will become more evident upon reading the follow- 
ing detailed description in conjunction with the accom- 
panying drawings, in which: 

[0010] Figure 1 is a block diagram of a system incor- 
porating a language identification module in accordance 
with the present invention. 

[0011] Figure 2 is a flow diagram of a method in ac- 
cordance with the present invention. 

DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

[0012] Referring to Figure 1 , a block diagram of a sys- 
tem with language recognition 101 in accordance with 
the present invention is illustrated. The system can be, 
for example, a messaging system capable of processing 
at least voice and/or textual commun Nations, and for 
purposes of this disclosure will be referred to as such. 
[0013] Messaging system 101 is connected to a net- 
work 102 for the receipt and transmission of communi- 
cations Network 102 can be an internet, intranet, ex- 
tranet, PSTN or any other communication network 
[0014] Messaging system 101 can access multiple 
mailboxes 103 that are assigned to specific users of the 
messaging system 101 . User mailboxes 103 can be lo- 
cated on separate systems. For example, e-mail mes- 
sage storage resides on a server distinct from a voice- 
mail system. Access to mailboxes 1 03 is provided to us- 
ers by means known in the art, such as computers 104 
or telephony equipment 105. 

[0015] Pursuant to the present invention, a communi- 
cation received by the messaging system 101 is ana- 
lyzed by a language identification module 106 to deter- 
mine its source language(s) by referral to language seg- 
ments (such as text segments, speech samples, and/or 
any other type of communication segment) tor a plurality 
of languages that are accessible from a database 106. 
The media type of the communication may dictate the 
type of language segments involved. 
[0016] A database 108 containing the language seg- 
ments may be resident to the system or accessible 
through the network 1 02. It may be necessary that a plu- 
rality of databases 109 be accessible by the messaging 
system 101 for the language identification module 106 
to be able to determine the source languagc(s) of a com- 
munication (or for subsequent language translation or 
conversion purposes). 

[0017] Language recognition/identification is 
achieved in the system by comparing arbitrary seg- 
ments of a communication to language segments for a 



plurality of languages (segment comparison). Prefera- 
bly, a segment of the communication of a particular me- 
dia-type is compared with one or more language seg- 
ments of the same media-type only. This need not be 
s the case, however, since a comparison of text segments 
to speech samples or vice-a-versa will not result in a 
match. Use of similar media-type only reduces the quan- 
tity of searching required. 

[0018] Use of a dictionary-type lookup (e.g., hash ta- 
10 bles, look-up tables, etc.) to identify the most likely lan- 
guage of a segment is central to identification (and sub- 
sequent processing features (e.g., translation, conver- 
sion, etc.) of the system). Database-stored dictionaries 
should contain a suitable number of common entries to 
15 facilitate identification. A language database 1 08, or plu- 
rality of databases 109, is (are) suitably configured to 
optimize searching of each dictionary for any segment 
(or possibly portion thereof). Additional dictionaries may 
also be used to enhance the accuracy of language iden- 
20 tification. 

[0019] Referring to Figure 2, a block diagram of a 
method in accordance with the invention is shown. A 
messaging system will be used for example purposes 
only and should not be taken as a limitation on the type 
25 of communication system for which the method can ap- 
ply. 

[0020] In accordance with the invention, a communi- 
cation is received by a communication system (e.g., 
messaging system) 200. If appropriate or necessary, the 
30 media-type of the communication is identified by means 
known in the art 205 (e.g., media-type identifiers trans- 
mitted within communication headers, based on the 
source (e.g., telephone, facsimile, computer, etc.)). It is 
possible that the communication system may only be 
35 utilized with speech only or text only communications, 
thereby eliminating the need to identify media-type. 
[0021] Once the media-type is known or identified, as 
appropriate, language segment(s) 210 are extracted 
from the communication. A language segment can be 
40 made up of one or more text or speech (or etc.) words 
(e.g., words contained within lines or sentences con- 
tained within a communication) when text is involved, or 
phonemes (e.g., phonemes as combined to create a 
spoken word or words) when speech is involved. Extrac- 
ts tion 21 0 of segment(s) from the communication can oc- 
cur one at a time or all together. 

[0022] For embodiments where segments are ex- 
tracted 210 from the communication all together, appro- 
priate dictionaries are searched 220 for language seg- 

so ments matching the extracted segments. Although seg- 
ments from the communication could be searched ran- 
domly, it is preferred that segments are searched 220 
systematically and/or sequentially. Segments arc 
searched 220 within the language dictionaries until it is 

55 established that the entire communication has been 
processed 230. 

[0023] For embodiments where segments are ex- 
tracted 21 0 from the communication one at a time, each 
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segment upon extraction is individually searched 220 
via the dictionaries. The method would continue with ex- 
traction 210 and searching 220 until the entire document 
has been processed in this way and sampled 230 before 
proceeding to scoring 240. In Figure 2. a feedback loop s 
is provided to the segment extraction step 210 from de- 
cision block 230 to show this. Again, it is preferred that 
searching of segments is conducted systematically and/ 
or sequentially. 

[0024] The outcome of the entire search (s) is scored io 
240 based on matching results for each individual seg- 
ment to identify the most likely language or languages 
contained within the communication and for determining 
an associated confidence level(s) for the identification 
of the language(s). Scoring could involve counting the is 
number of 'hits" for each segment against all dictionar- 
ies until a suitable statistical confidence level is 
achieved. Applying a weighting factor to segments en- 
hances recognition performance, because some seg- 
ments may very clearly and uniquely identify a specific 20 
language and hence only a very few segments will re- 
quire examination. Extracted segments can be passed 
through the search mechanism during recognition until 
a sufficiently high confidence level is attained. Heuristics 
may be applied to remove constructs that arc unlikely ss 
to aid in identification (numerics, obvious abbreviations, 
etc.). It is by thoroughly sampling the entire communi- 
cation 230, that it can be determined when an author of 
the communication may have changed languages. 
[0025] Once the language(s) is/are identified 250, 30 
segments of the communication can be marked for iden- 
tification 260 for further processing 270 (e.g., transla- 
tion, conversion, archiving, transmission, rendering; 
etc). A pre-processing step 280 may be used to apply 
heuristics that recognize specific prefix and suffix pat- 35 
terns. A set of common usage words can be used for 
identification purposes, along with their common prefix- 
es and suffixes. Such data is suitably stored in, for ex- 
ample, hash tables to optimize searching. One or more 
hash tables may be used. Pre-processing may be im- 40 
plemented alone or concurrently with media-type iden- 
tification, as shown in Figure 2. 

[0026] After the language has been identified, the sys- 
tem may continue to look up each segment against its 
dictionary. Allowing for expected segment errors (e.g., 45 
misspellings in the case of text, mispronunciations in the 
case of speech, etc.), if the number of "misses" be- 
comes excessive, language identification may be re- 
peated to determine if another language is now more 
appropriate. Identification with the new language may so 
begin at the point in the communication where the miss- 
es commenced. The new language may be applied, ret- 
roactively, to the beginning of, for example, a sentence 
or paragraph, or spoken sentence or phrase where the 
misses were noticed. ss 
[0027] Referring again to Figure 1 . in accordance with 
a further aspect of the present invention, a language 
translation module 107 can be provided for the transla- 



tion of the communication (e.g., received message in a 
messaging system), and in particular, the segments 
contained therein from the identified source language 
into a secondary (foreign) language or languages. The 
translation module 1 07 accesses the database 1 08 and/ 
or databases 109 during translation operations to ena- 
ble such translation. US 5,768,603 and US 5,765.131, 
which are herein incorporated by reference, discuss ex- 
amples of language translation which could be used by 
the present invention. The translation module 107 and 
database(s) 108/109 can accommodate translation nu- 
ances (e.g. language syntax and formation differences) 
based on the translation option (e.g., translating from 
Spanish to English). 

[0028] Clearly, it is necessary for the language iden- 
tification module to do more than simply identify the lan- 
guage of the communication for language translation 
applications. If more than one language has been found, 
the text must be partitioned into multiple segments and 
a language identifier applied to each. The translation 
module 107 can then process each part separately, uti- 
lizing the appropriate language "dictionary" for each in- 
dependently identified segment. 

[0029] In accordance with still another aspect of the 
prosont invention, a tcxt-to-spooch conversion module 

110 can also be incorporated into the system 101 for 
use with text communications, to provide for conversion 
of the communication from text-to-speech. The text-to- 
speech module 110 can access the language database 
1 08 or databases 1 09 to convert the communication into 
speech in the source or secondary language. The text- 
to-speech module 110 can convert the communication, 
either before or after being translated, into speech for 
retrieval by the recipient from the network 102. US 
5,774,854 : which is herein incorporated by reference, 
describes a method for performing text-to-speech con- 
version. The text-to-speech module 110 works with the 
language identification module 106 to identify the 
source language(s). This is necessary to know what lan- 
guage database(s) to use, and is done by reference to 
text segments contained within databases 108/109. The 
databases 108/109 will include a subset (i.e., text seg- 
ments) of more than one language for the purposes of 
not only carrying out initial text language recognition, but 
also text-to-speech conversion. 

[0030] In accordance with yet another aspect of the 
present invention, a speech-to-text conversion module 

111 can be included in their system 101 for use with 
speech communications, to provide for conversion of 
the communication from speech to text. The speech-to- 
text module 111 can access the language database 108 
or databases 1 09 to convert the communication into text 
in the source or secondary language. The specch-to- 
text module 111 can convert the communication, either 
before or after being translated, into text for retrieval by 
the recipient from the network 102. The speech-to-text 
module 111 works with the language identification mod- 
ule 106 to identify the source language(s) This is nec- 
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essary to know what language database(s) to use, and 
is done by reference to speech segments (i.e., speech 
samples (e.g.. phonemes)) contained within databases 
108/109. The databases 108/109 will include a subset 
(i.e., speech samples) of more than one language for 
the purposes of not only carrying out initial speech lan- 
guage recognition, but also speech-to-text conversion. 
[0031] Once in text form, the text communication can 
be accessed by the user in textual form from a desktop 
104 (e.g., as e-mail) or translated by the translation 
module 107 into another language. After translation, the 
message can be then be accessed by the desktop 1 04 
or converted back into a voice message utilizing the text- 
to-speech module conversion 110. If converted to 
speech, the message can be retrieved via telephony 
equipment 105 or computer 104 by accessing messag- 
ing mailboxes 103. US 5,724,410, which is herein incor- 
porated by reference, teaches a method for two-way 
voice-messaging capable of speech-to-text conversion 
where the source language is already known. 
[0032] Continued access to a database 108, or plu- 
rality of databases 109, of dictionaries for languages of 
interest is required for the execution and completion of 
translation module 107 and text-to-speech module 110 
or spccch-to-tcxt modulo 111 operations. Once the lan- 
guage identity for a segment (or segments) within a 
communication has been identified, the language dic- 
tionary from appropriate databases can be used to effi- 
ciently carry out translation or conversion of a commu- 
nication 

[0033] It should be appreciated that all the compo- 
nents of the recognition system are not required to be 
resident in a messaging system 101 as depicted in Fig- 
ure 1. For example, the language identification module 
106 and mailboxes 103 can be networked as the plural- 
ity of databases 109 are. It should also be appreciated 
that other media-type conversion modules can be inte- 
grated with the present system (e.g., Teletype for the 
Deaf and Disabled (TDD)-to-speech, to-text, vice-a-ver- 
sa, etc.) 

[0034] The above-described system can be imple- 
mented as processor-driven software designed to com- 
pare an arbitrary language segment from a communica- 
tion against segments for a plurality of languages stored 
in at least one database in order to achieve a linguistic 
match, and thereby identify the communication as being 
of particular source language. The communication 
could then be translated into another language or con- 
verted from text into speech or speech into text in the 
identified language, prior to retrieval by a recipient. 
Translating the communication from the source lan- 
guage into selected secondary languages or conversion 
from one media-type to another is conducted by provid- 
ing the computer with access to source/initial media- 
type and corresponding secondary/media-type lan- 
guage segments from a database. 
[0035] In general, the invention may be applied to any 
communication that is processed by any communication 



system. Any emphasis placed upon multimedia mes- 
saging systems or messages, such as e-mail or voice- 
mail, pertains only to one particular embodiment of the 
invention. 

5 [0036] It should be appreciated that a communication 
can be converted from one media type (e.g., speech-to- 
text) to another prior to translation, and subsequent to 
such translation, converted back to the original media 
type (e.g. text-to-speech). 

io [0037] It should be appreciated that a recognized 
communication can be played back in speech form on 
a desktop. Such a capability would be every useful, for 
example, in assisting the reading disabled. It should also 
be understood that generated speech could be stored 

7£ on a disk as a file or mail message, or as an attachment 
to a maif message and so on. A user could then listen 
to speech at a later time, or even send the audio speech 
to other persons that may not have access to the original 
communication, or translation/conversion facilities. For 

20 example, a message in a multiplicity of secondary lan- 
guages can be sent to others within a multipart/alterna- 
tive e-mail encapsulation. An appropriately equipped 
desktop 1 04 could then automatically play the document 
in the user's language of choice. 

2£ [0038] It should also be appreciated that text forms of 
a text or speech communication can be transmitted via 
facsimile. A such, this will enable a communication 
which was originally received by a communication sys- 
tem in accordance with the invention, regardless of 

30 where it originated from or in what language to be re- 
trieved via a facsimile device subsequent to translation 
and/or conversion. 

[0039] While various embodiments of the present in- 
vention have been described above, it should be under- 
35 stood that they have been presente'd by way of example, 
and not limitation. Thus the breadth and scope of the 
present invention should not be limited by any of the 
above-described exemplary embodiments. 

40 

Claims 

1. A communication system, comprising a language 
identification module for automatically identifying a 
4 $ source language of at least one arbitrary segment 
of a communication by comparing said arbitrary 
segment to one or more language segments for a 
plurality of languages. 

50 2. The system of claim 1 , further comprising a trans- 
lation module for translating said communication 
from said source language to at least one second- 
ary language. 

55 3. The system of claim 1 wherein said communication 
is in text form, and said system further comprises a 
text-to-speech module for converting said commu- 
nication to speech form based on said source lan- 
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guage. 



said communication. 



4. The system of claim 2 wherein said communication 
is in text form, and said system further comprises a 
text-to-speech module for converting said commu- 
nication to speech form after being translated into 
said at least one secondary language. 

5. The system of claim 1 wherein said communication 
is in speech form, and said system further compris- 
es a speech-to-text module for converting said com- 
munication into text form based on said source lan- 
guage. 

6. The system of claim 2 wherein said communication 
is in speech form, and said system further compris- 
es a speech-to-text module for converting said com- 
munication into text form after being translated into 
said at least one secondary language. 

7. The communication system of claim 1 , further com- 
prising at least one database containing language 
segments for at least one language, wherein said 
identification module accesses said at least one da- 
tabase during said identifying. 

8. The communication system of claim 1 , wherein said 
system is a messaging system. 

9. The communication system of claim 8, wherein said 
communication is a text message, and said system 
further comprises a text-to-speech module for con- 
verting said text message into speech form. 

10. The system of claim 8, wherein said communication 
is a voice message, and said system further com- 
prises a speech-to-text module for converting said 
voice message into text form. 

1 1 . The system of claim 8,9 or 1 0 wherein said commu- 
nication is accessible via a personal computer, a 
facsimile apparatus, a TDD, or a telephony appara- 
tus. 

12. The system of any preceding claim, wherein the 
identification module determines said source lan- 
guage by scoring results of said comparing. 

1 3. The system of claim 2, wherein said communication 
is received by said communication system in 
speech form, said system further comprising a 
speech-to-text module for converting said commu- 
nication to text form prior to said translating, and a 
text-to-speech module for converting said commu- 
nication to speech form after said translating. 

1 4. The communication system of claim 1 ,3 or 5, further 
comprising a means for determining media-type for 
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1 5. The communication system of claim 1 , wherein said 
language identification module is for automatically 
identifying multiple languages contained within a 
communication by comparing arbitrary segment of 
said communication to one or more language seg- 
ments for a plurality of languages. 

16. The communication system of claim 2, further com- 
prising a means for identifying media-type for said 
communication and a media conversion module for 
converting media-type of said communication. 

17. A language identification method for use by a com- 
munication system, comprising: 

upon receipt of a communication, automatically 
comparing at least one arbitrary segment from 
said communication with one or more language 
segments of a plurality of languages; and 
identifying said communication as being of a 
particular source language based on said com- 
paring. 

18. The method of claim 17 wherein said communica- 
tion is in text format, and said method further com- 
prises converting said communication from text for- 
mat into speech format based on said source lan- 
guage. 

19. The method of claim 17 wherein said communica- 
tion is in speech format, and said method further 
comprises converting said communication from 
speech format into text format based on said source 
language. 



20. The method of claim 17, further comprising trans- 
lating said communication from said source lan- 
40 guage into at least one secondary language, where- 
in said translating is facilitated by accessing at least 
one language database. 



45 



SO 



21 . The method of claim 20 wherein said communica- 
tion is in text format, and said method further com- 
prises converting said translated communication 
from text format into speech format. 

22. The method of claim 20 wherein said communica- 
tion is received in speech format, and said method 
further comprises converting said communication 
from speech format into text format based on said 
source language prior to said translating. 



55 23. The method of any of claims 1 7 to 22, wherein said 
comparing is of an arbitrary segment from a com- 
munication received into an electronic multimedia 
system against language segments of a plurality of 
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31. The method of claim 30, wherein said message is 
converted into a second media type after said trans- 
lation for retrieval in said second media type. 



10 



language databases to achieve a linguistic match; 
and wherein said identifying is of said communica- 
tion and is for identifying said communication as be- 
ing of a particular source language based on said 
comparing. 

24. The method of claim 23, comprising accessing said 
converted communication via a telephony appara- 
tus or a personal computer. 

25. The method of claim 17, wherein said identifying in- 
cludes scoring results of said comparing. 

26. The method of any of claims 1 7 to 22, wherein said 
comparing is of an arbitrary segment from a mes- is 
sage received into a messaging system against lan- 
guage segments of a plurality of language databas- 
es to achieve a linguistic match; and wherein said 
identifying is of said message and is tor identifying 
said message as being of a particular source Ian- 20 
guage based on said comparing. 

27. The method of claim 26, comprising accessing said 
converted message via a telephony apparatus or a 
personal computer. 25 

28. The method of claim 26, wherein said translation 
into said at least one secondary language is exe- 
cuted automatically based on a message recipient's 
preselected mailbox preferences specified within 30 
said messaging system for said recipient. 

29. The method of claim 22, further comprising the step 
of converting said communication after said trans- 
lating from text format into speech format. 35 

30. The method of claim 26, wherein said message is 
received in a first media type and is converted into 
a second media type prior to said translating. 
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